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■ Abstract 

o : 

In this paper wc study the dynamical instability of Sakai-Sugimoto's holographic QCD model 
at finite baryon density. In this model, the baryon density, represented by the smeared instanton 
on the worldvolumc of the probe D8-D8 mesonic brane, sources the worldvolume electric field, 
and through the Chern-Simons term it will induces the instability to form a chiral helical 



wave. This is similar to Deryagin-Grigoriev-Rubakov instability to form the chiral density wave 
for large N c QCD at finite density. Our results show that this kind of instability occurs for 
sufficiently high baryon number densities. The phase diagram of holographic QCD will thus 
be changed from the one which is based only on thermodynamics. This holographic approach 
provides an effective way to study the phases of QCD at finite density, where the conventional 
perturbative QCD and lattice simulation fail. 



1 Introduction 



The quantum chrome-dynamics (QCD) at high baryon density has implications for physics in many 
fields, such as the possible existence of quark matters in the core of compact stars. However, 
unlike the physics for QCD at finite temperature, it is difficult to directly probe such a regime 
theoretically, even from the first principle numerical simulations. This is mainly due to the complex 
fermion determinant at finite density, which causes the sign problem in simulation. 

Despite that, it was conjectured that there is a color superconducting ground state for QCD at 
high enough density [I], similar to the BCS mechanism for the formation of Cooper pairs of quarks 
near the Fermi surface. Since the Fermi momentum plays the role of an additional energy scale, we 
can apply the perturbative techniques in the regime of ultra-high density. However, at moderate 
density where the most interesting physics dwells, QCD is strongly coupled and it is hard to have 
reliable treatments. 

Instead, one can consider QCD at large N c (number of color) limit to probe the above non- 
perturbative regime of finite density. This can be done either by using the renormalization group 
improved perturbative technique for small ' t Hoof coupling, or by studying the holographic dual 
gravity for large 't Hooft coupling. For the former case, Deryagin, Grigoriev and Rubakov (DGFl) [2] 
noticed that color superconductivity will be suppressed in the large N c limit since the Cooper pair 
of quarks is not color singlet and its formation is diagramtically non-planar. Moreover, they found 
that there is a new dynamical instability for the formation of chiral density wave whose wave 
number is the twice of the Fermi momentum [2j[3], i.e., a spatially modulated phase. This can be 
understood as the standing wave from the formation of quark-hole pairs. 

On the other hand, the study of the gravity duals of large N c M = A Super- Yang-Mills(SYM) 
or QCD are shown to be able to capture many essential features of real QCD at high tempera- 
ture and finite density such as what happened at RHIC. Indeed, a pioneering work by Domokos 
and Harvey[l] has shown that the bulk Chern-Simons term will induce a dynamical instability to 
form the DGR-like modulated phase in the zero temperature holographic QCD from a bottom-up 
approach. Recently the interest in the dynamical instability induced by the Chern-Simons term 
is revived by Nakamura, Ooguri and Park in [5]. They have explicitly shown that the new mod- 
ulated phase is the helical wave of a holographic U(l) current. These motivate us to study the 
same issue in the top-down approach of the holographic QCD, i.e., Sakai-Sugimoto model[6l[7] at 
finite temperature |8j and finite density [13]. The advantage of the top-down approach enables us to 
understand how marginal the above dynamical instability is in a more consistent and systematical 
way. In our model we figure out a window for modulated phase of chiral helical wave at high 
temperature C8.SC (\S shown in Fig. [TJ 

In the next section, we will briefly review the holographic QCD proposed by Sakai and Sugimoto[6l 
[7], and then study thermodynamics of its generalization to the case with finite temperature and 
finite baryon density [15]. In Sakai-Sugimoto model the meson is dual to a pair of connected D8 and 
D8 branes, and the baryon is dual to DA branes wrapping on the internal S* 4 [9]. There is a pulling 
force on D8 exerted by the wrapped DA whose strength is proportional to the baryon density. This 
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geometrizes the change of QCD vacuum by the presence of finite baryon density. In section 3 we 
discuss the dynamical instability in holographic QCD and derive the master equation for it, and 
in section 4 we discuss our numerical analysis to find out such an instability. The physical result 
of our analysis is summarized in the phase diagram as shown in Fig. [TJ Finally, we conclude the 
paper in section 5. 




-1.5 -1 -0.5 0.5 



Figure 1: Phase diagram of our holographic QCD model at finite temperature and finite baryon 
density. The horizontal dashed line around T = 0.075 separates the low temperature (LT) and 
the high temperature (HT) phases. The left two purple lines, almost vertically straight in both of 
LT and HT phases, separate the nuclear matter phase and the vacuum phase where the baryon 
density vanishes. The horizontal curved blue line in the high temperature phase, starting from 
T ~ 0.14, is the phase boundary between the nuclear matter or the vacuum phase, and the QGP 
phase. This line is determined thermodynamically. The three red lines on the right, inside the 
nuclear matter phase of both LT and HT phases and also in the QGP phase, denote the onset of 
dynamical instability in each phase. On the right of these lines, the system may be in a spatially 
modulated phase, though we do not discuss the stability of this modulated phase in the paper. The 
parameters (whose meaning will be clear in Section [2] and EJ) are set to be R = 1, L = 0.53 and 
K ~ 1.018. 
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2 The Model of Holographic QCD 



The Sakai-Sugimoto(SS) model consists of Nf pairs of D8 and D8 branes embedded in the near 
horizon geometry of N c DA branes on a Scherk-Schwarz circle, which completely breaks the su- 
persymmetry. The size of the circle basically controls the mass scale of the unwanted fields. The 
configuration can be summarized as follows: 

0123 (4) 56789 
iV c DA x x x x x (1) 
Nj D8D8 xxxx xxxxx 

The x 4 direction is the Scherk-Schwarz circle with the anti-periodic boundary condition for fermionic 
fields. 

The probe D8 branes have the embedding function X4 = xa(U) and extend in the other 8 
dimensions, where U is the radial coordinate of the background DA brane geometry. The embedding 
function can be determined by solving the equations of motion of the D8 DBI action. In [6l [7] 
the authors found that there is a smooth interpolation of the D8-D8 pair from the DBI action 
analysis, which was then interpreted as the chiral symmetry breaking of QCD. In this phase, the 
background geometry is described by ([3]), the dual effective QCD is confined[8] and the degrees 
of freedom are hadronic. By changing the time direction to a thermal circle, we can introduce 
temperature into the model. At a critical temperature, the geometry has a phase transition and 
becomes another geometry with horizon [HJ [12]. This is the high temperature phase that corresponds 
to the deconfined phase of QCD and will be described in section 12.21 in detail. 

In AdS^/CFT^ correspondence, the baryons are the Dh branes wrapping on the S 5 . The RR- 
flux is coupled to the gauge field on the D5 worldvolume and becomes a source for the gauge field. 
In order to cancel the tadpole, there will be N c strings attached to the D5 which looks like a 
pointlike particle in ^4d5s0. Similar constructions of holographic baryons were done for SS model, 
which is carried out by wrapping DA branes on the internal S 4 \W\ [TT] , and further smearing the 
DA charges in the non-compact directions for the discussions of QCD phase diagram |13j. The DA 
charge density then plays the role of baryon number density in the dual QCD. From the point of 
view of the probe D8, the wrapped D4's are also regarded as instantons on the D8 worldvolume. 
However, it turns out that the complete treatment of the baryons as the world- volume instantons 
is quite difficult and is yet unknown. Thus we include the energy contribution of the instanton as 
D4-brane DBI action, which was done in [13|. Below we will review this results that will provide 
the background configuration for our further study of modulated phase instability. 

After wrapping the DA, we also have to include the Chern-Simons(CS) action on the D8 brane 

Scs = T 8 J C p +i A tr exp(2WF) (2) 

if there is non-trivial RR potential C p +i generated by Dp-brane or non-trivial gauge curvature F 
on the D8 worldvolume. We will regard the wrapped DA branes as sources to generate non-trivial 
F along 1,2,3, U directions and focus on the coupling of U(l) part to SU(Nf) part of the gauge 
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field on D8 brane. The SU(Nf) part of the gauge field is the anti-self dual connection whose second 
Chern character is the number of D4 branes, i.e., g-ytr F 2 = Nfn±8(U — U c )d 3 xdU , where is 
the dimensionless instanton number density and we also take the total number of the instanton to 
be a multiple of Nf. 

2.1 Thermodynamics of the low temperature phase 

We first consider the low temperature (LT) phase of QCD in which the quarks and gluons are 
confined, and the chiral symmetry is spontaneously broken. The DA background solution dual to 
the low temperature phase of QCD is given by |12j . 

ds 2 = Q 1 (T i p,dx»d X v + h{U)dx 2 ) + (|) 1 + U 2 d& 

„ (2n) 3 l 3 N c 6 (U\\ LfTTS , Ul K 4ttR 3 / 2 

F *= ^ e =g \R) ' h W = 1 -ijf> ^-^4 + —^, (3) 

6U KK 

where fj,, v = 0, 1, 2, 3, U is the radial direction, the dVt\ and €4 are the metric and volume form of 
the S" 4 of unit radius, and R 3 = ng s N c £ 3 . This geometry has a cigar shape terminated at U = Ukk- 
The DBI action and CS action in terms of the D8 brane embedding function x±{U) and the 
electric field E(U) := —A' are the following 

Sdbi = -N j dUU^hxf + (l/U) 3 (l/h - A' 2 ) , (4) 
S cs = ^2 I A ° trp2 = Nn b I dU6(U - U C )A , (5) 

71 " JM 4 xR+ J 

where the ' is the derivative with respect to U, (3 := f dxo is the inverse temperature, V3 
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J dx\dx2dx3, Q4 is the volume of unit <S 4 , ./V := TaR N f V3 ^ n4 ; anc i the dimensionless baryon number 
density is n& := ~^^r^jf- ■ Here, for the DBI action, we only include the U(l) part of the gauge 
field and neglected the contribution from the SU(Nf) parto The contribution from the SU(Nf) 
part is basically the energy of the instantons and this will be included as D4-brane action on S 
when the stationary condition is considered. The D4-brane action on 5 4 at U = U c is given by 

Nnb TT (a\ 
= ^-U c , (6) 

u=u c * 

where N4 = Nfn^V^/ R 3 is the number of D4-branes. 

One can derive the equations of motion from @, and then solve for X4(U) and E(U): 



Sd4 = — N4T4 J dQ/^dre * \J — det g 



x > = ±U - 2 - E = -n b U2 -. , 7 

h s/H - H sin 2 C V H - H sin 2 C 



In the following we re-scale the coordinates (U, xo, xi, X2, 2:3) by factor of R so that they are dimensionless, and 

\/2NfR 

also rescale U(l) part of the gauge fields by factor of 2 -wa' — ■ ^^ so n °te that these actions describe the upper half of 
the connected D8-branes, and then the actual action is twice of this. To investigate the phase structure, it is sufficient 
to look at this half and we will consider only this part of the action. 

2 It is not difficult to see that the contributions from this non-Abelian part is completely mixed, in a complicated 
way, with the Abelian part functions we will now solve, and also the complete form of the contribution is yet unknown. 
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where 

H(U) = U 3 h(U) (U 5 + n 2 b ), H = H(U C ), tan 6 C = y/g^/guu <\u=u c , (8) 

where 544 and guu refer to the corresponding factors in the metric Q. 
To further solve ([7]), we need to impose the fixed length condition, i.e., 

POO 

L= dUx' 4 , (9) 



where 2L is the UV separation of D8 and D8 along ^-direction. The D8-D8 profiles for zero and 
non-zero are shown in Fig. [2j Note that this parameter L is particular to the SS model and does 
not correspond to any physical parameter in the real QCD. We will set L = 0.53 and Ukk = 0.1 
throughout our analysis. This choice is partly motivated by the physical parameter choice of SS 
model and will be explained more in Section [H 

Geometrically, the above solution arises from the balance between the D8's tension from the 
DBI part and the pulling by DA wrapping on S 4 . As a result, the D8 profile develops a cusp at 
IR end U = U c with a cusp angle 9 C appearing as a constant of integration in ([7]) and ([8]). This is 
determined by the force balance solution 

1 <m Bt+ id^ ) 



^gUU dU c yJgUU dU c /on-sholl 

where Ho$ = —Sdbi + / dUUA A' , denotes the Euclidean D4 DBI action with n^ the 
conjugate momentum of Aq, and the total derivative is defined as -^jj- := Jjjj- + (^-§jj-^- u=u 

n b pi fixed 

fo^rjjjj which should be performed before imposing the on-shell condition. Solving (jlOjn for the 
cusp angle 6 C yields 

co^e c = h{Uc) (ii) 

Note that as reduces to 0, 6 C = ir/2 and the cusp becomes a smooth tip. 

For a given nt,, the cusp location U c can be determined by solving the fixed length condition 
([9]). This relation is displayed in Fig. [3](a). As rib increases, D4-brane pulls the cusp towards the 
tip of the cigar-like space, but at some point the tension of D8-brane overcomes the pulling force 
and the position of the cusp goes to large U region. 



3 To solve it, one may need to use the trick pointed out in the footnote 6 of [14] 
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Figure 3: The cusp position U c against log 10 n b at Ukk = 0.1, L = 0.53 for (a) the low temperature 
phase; and (b) and (c) are the high temperature phase at T = 0.0755 (or Ut = 0.1) and T = 0.1997 
(Ut = 0.7), respectively. (See sec. 2.2). 



Next, we discuss the thermodynamics of the D8-D8 branes. We will neglect the back reaction 
from the probe branes so that the contribution to the free energy from the DA background geometry 
will be the same for different configurations of probe branes, and can be omitted and considered 
separately. Therefore, the grand canonical potential of holographic QCD is given by |16| 



fiz) 8 (T, fj.) 



1 

N 



S 



DBI I on— shell 



+ s, 



I on— shell / ' 



(12) 



where E denotes the Euclidean action, while the Helmholtz free energy if" 

1 



F D8 (T, n b ) 



N 



S 



^S/lcm-shell 



+ n b 



poo 



dU 




h{u){x' A y + 



U 3 h(U) 



nr. 



(13) 



x 1 ^ on— shell 



where ji is the chemical potential defined by 



jJL := Aq(U ->■ oo) 



(14) 



Note that the temperature dependence in the IT phase is trivial and the following expressions have 
actually no T dependence in this phase. The system contains also wrapped D4-branes, and its free 
energy is simply given by the Euclidean action (normalized by the factor N). Thus the total free 
energy is 



F{T,n b ) =F D8 (T,n b ) + ±SE i . 



(15) 



4 In the second line of ()13p . A' had been replaced by x\ by means of the equation of motion. Here we choose to 
express F{T,rib) by 14 for the convenience of calculating the chemical potential jj,. In the following, (|16l) is carried 
out as a chain-rule differentiation since F here is a on-shell quantity. Equivalently, one can also treat F as off-shell 
and use the functional variation method to derive /x. The result is the same. 
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From thermodynamics, the chemical potential is also obtained by 



dF 
dn b 

dF D8 



T,L 



dn h 



+ 



dF D8 dx' A 



x' 4 ,U c ,T,L 



dx' 4 dn b 



+ 



(dFps | 1 dS% A \ dU r . 



U C ,T,L 



\dU c N dU c J dn b 



+ 



1 dS* 4 



x' A ,T,L 



N dn b 



x' A ,T,L 



(16) 



Note that this formulation is under the implicit constraint of L being fixed. The second term on 
the right hand side vanishes, because J-p- gives rise to an constant of motion inside the integral, 

and Iu c dnt ~ dn b ~ since L is fixed |13|. The third term also vanishes due to the force balance 
condition (jlOh (note that NFds = Hds)- The remaining terms are consistent with the fixed length 
condition. As a result, the chemical potential reads [§ 



1 B C I E 
N on b 



TJ3 

dll rih\ / — - — ^ — 

Uc b \H(U)-H c sm 2 9 c 



+ 



Ur 



(17) 



The chemical potential ri as a function of n b is displayed in Fig. H^a). As n b goes to zero, ll 
approaches a nonzero value. Below this value, there is no cusp configuration and then the system 
is described by the D8-D8 configuration without instantons. This is called the vacuum phase [T3] . 
In this phase, ii is a free parameter and we compare the grand canonical potential of the nuclear 
matter phase and the vacuum phase, and conclude that as long as the cusp configurations exist, 
the nuclear matter phase is always favored. 



(i 




rih 

(a) (b) (c) 

Figure 4: The chemical potentials versus n b in: (a)the low temperature phase, (b) the high temper- 
ature nuclear matter phase at T = 0.0755(See sec. 2.2) and (c) the QGP phase at T = 0.0755(See 
sec. 2.2). In all the figures, the horizontal line denotes n b and the vertical one ji. 



5 c.f. footnote 8 of [T4l 
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2.2 Thermodynamics of the high temperature phase 

Similarly, one can also consider the high temperature (HT) phase of holographic QCD, which is 
dual to the following metric [8] 



'[A 3/2 /iA 3/2 r 
ds 2 = (— J [-h(U)dt 2 + dx\ + dxl + dxj + dxl]+ ( — j 



du + u 2 dn 2 



h(U) 

h(U) = (18) 
The dilaton and the 4- form flux are the same as in ([3]). The Hawking temperature determined from 

_3_ [Th 
4tt V 7F 



the metric is T := t^t which is the temperature for holographic QCD. 

The transition between the LT and the HT phases occurs at T c = ^y^p^ — 0.0755 [8]. 
Above T c the temperature is varied by tuning Ut, and the dual QCD is deconfined. This critical 
temperature is obtained by comparing the free energies of the supergravity background of each 
phase. This is the background geometric transition which happens in the leading order at large iV c . 
Since the effect from the Nf quarks can be neglected in the planar limit, the boundary between 
the HT and LT phases is straight along the axis of ji, as shown by the dashed line in Fig. [TJ 

Embedding the D8 and D8 branes in (|18h . by the same procedure in the LT case we can solve the 
D8-brane profile and the electric field, now together with the D4-brane action = ^/ h(U c )U c . 
The cusped solution is given by 



x' 4 = ±U^ 2 J - H ° sin2e c E = -n b U*' 2 J- „ (Nuclear matter) . (19) 

4 Y h(H- H sin 2 9 e ) y (H - H sin 2 6 C ) y ' V ' 

The function H and the angle 9 C are also given by ([8]) and (jlip . with h taken from (|18p . Since 
the D8 and D8 are joined at U c with a cusp angle 9 C , with broken chiral symmetry this solution 
describes the nuclear matter in the dual QCD. Variation of U c with n& at T = 0.0755 (i.e., Ut = 
0.1) and T = 0.1997 (Ut = 0.7) are displayed in Fig. E[b) and (c). For lower temperature, the 
behavior is similar to that of the low temperature phase, and there always exists a solution. For 
higher temperatures, there will be no solution of the fixed length condition for small rib, but these 
temperatures turn out to be in the QGP phase, which is described below, and then we will not face 
this missing solution problem. 

The grand canonical potential, the Helmholtz free energy, and the chemical potential in the HT 
nuclear matter phase can be obtained formally from (|12p , the first line of ()13|) and (|15p , and ()16p re- 
spectively. The chemical potential is given by // = A' dU = f™ c dU n b\J jj^^^i 9c + 
Uc^/h(Uc) ^ rp ne b enav j or ^ a g a i ns t nb i s displayed in Fig. HJb). Again, fi goes to a finite value as n& 
vanishes, and there is also the vacuum phase in the high temperature region. 

Moreover, unlike the LT case where the parallel D8-D8 configuration is not allowed unless 
L = 0, in the HT phase we may have either the the chiral symmetry broken configuration (|19p . or 
the chiral symmetry restored one described by the parallel D8 and D8: 

x' 4 = 0, E = -A' Q = -j^- (QGP). (20) 

U 5 + n 2 
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Both branes extend in the radial direction from the horizon Ut to the boundary. Since the chiral 
symmetry is restored and the effective theory is deconfined, the configuration (|20l) describes the 
QGP phase. The chemical potential of the QGP phase is 

/•oo 

ii:= j A f dU = A (U -»• oo) , (21) 
Ju T 

where Aq(Ut) is set to on the horizon to satisfy the regularity requirement. In the QGP phase 
there is of course no baryon degrees of freedom, so here we should regard the chemical potential 
as that of the liberated quarks and anti-quarks from the dissociation of baryons and mesons. Th 
chemical potential as a function of rib is displayed in Fig. 0(c). For this QGP case, fi vanishes as 
rib goes to zero. 

To obtain the boundary between the nuclear matter and the QGP phase, we compare the grand 
canonical potential Q, which are Q x b(T, fi) = dUU 5 H ^ fj^m^ g ~ ^ or ^ e nuclear matter phase 
and Oqgp(T, fj) = dU ■= = for the QGP phase. Note that the each rib in the above formulae 
takes different values in each phase, for the same T and \i. From our numerical calculation, we 
determine the phase boundary where the difference of the potential goes to zero. These are the 
boundary between the QGP and the nuclear matter in the high temperature phase in Figure 1. In 
this high temperature phase, there is also the vacuum phase where the D8-brane does not develop 
the cusp. The boundary between the vacuum and the nuclear matter phase is determined by 
lim nb _ s .o fJ>{n>b) f° r a given temperature, and by comparing the grand canonical potential the nuclear 
matter phase is again preferred if exists. The boundary between the QGP and the vacuum phase is 
also determined by comparing the grand canonical potential. The potential for the vacuum phase 
is given by O vac (T) = dU yj ^^^{^ y 

3 Dynamical Instability 

To see if there is any dynamical instability of holographic QCD at finite baryon density, we need to 
consider the bulk fluctuations around the above background. We then solve the bulk fluctuations 
and derive the expectation values of the corresponding operators. If the expectation value grows 
with time, it suggests the vacuum is unstable and implies dynamical instability. 

For our purpose, we turn on the fluctuations U) for D8 profile and fu(x^, U) := diaj — 
djaj for D8 gauge fields. Holographically, the fields Oj, ay and y are dual to the chiral current 
(Jl R ), pion field (qj 5 q} and chiral symmetry violation (d r j£ ^)[18j. respectively. If there are 
normalizable unstable (i.e., its magnitude grows with time) solutions for these bulk fluctuations, it 
implies the dynamical instability to new phases. For example, if exists, the modulated phase for 
ctj, as discussed in [5], is dual to helical structure of large _/V c QCD; and the new modulated phases 
for ajj and y will be dual to the chiral density wave, a kind of DGR instability. 

To obtain the field equations for the above bulk fluctuations, we expand the Lagrangian Ldbi + 
Lcs U P t° quadratic order for both low and high temperature phases, and perform the corresponding 
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variations. The results are 



t {t^ - (1 - * - V4w«} -\j^{y'- t Li ) }' = ' (22) 

£ l-A p x' 4 E dfy + (a p (4) 2 + dif 0i \ + |^ (f ou - \ 1=0, (23) 

?- Wot/ - - #d Li) = , (24) 



-Ke ijk (2A djf kU - Ef jk + A duf jk ) = , (25) 



where Lq and Li represent the expressions 



L =U 4 Jh(x> 4 )2 + [7-3 ^ J_ _ ^ ; Ll = g _ ^-3^ /oc/) 



(26) 



and the index p labels different phases for the factor A p , such that Alt '■= h and Aht := 1. Note 
that the formulation of /i in the LT and HT cases should be correspondingly taken from ([3|) and 
(JT8J) respectively. 

Though these are coupled equations which cannot be decoupled in general, we can isolate the 
equation for /j := \eij k fjk from the other bulk fluctuations by applying e%j k dj to (|25j) . We then 
arrive at the master equation 



U b h A Lq 



(l + f^ 5 ) 3 2 / fc + «9 2 A.j + 2ne ijk Ed j f i = , (27) 



in which A p (x^) 2 + = ^ + |^ is used. Note that y, ajj and ao are decoupled. This is 
the similar equation discussed in [5] for the instability to form the modulated phase; however, the 
effective CS coupling is not a fixed constant as in [5] but given by 

and thus it depends on the choice of the parameters like 't Hooft coupling A := <?y A:f iV c , but not on 
the instanton number density. 

The parameter k and the baryon number density rib will turn out to govern the strength of 
instability to form the modulated phase as discussed in [5]. So when rib and k take a sufficiently 
large value, the dynamical instability would appear, as we will confirm by numerical calculation in 
the next section. 
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4 Numerical Results 



In this section, we present our numerical analysis for possible dynamical instability, and the physical 
result is summarized in the phase diagram shown in Figure 1. 

We start by summarizing our choice of the parameters. We set Mkk = § \J an d 't Hooft 
coupling A = gy M N c as Mj^x — 950MeV and A ~ 16.71 (for example, [IS]). This choice was set by 
using p- meson mass and the pion decay constant. We still have a freedom to choose the mass scale 
by R and then set Ukk/R = 0.1 and all the dimensions are adjusted by including R appropriately. 
This choice leads k ~ 1.017980206, and we choose the asymptotic half separation L = 0.53 0. 

We numerically solve the equations of motion derived in the previous section to find unstable 
modes. We use the standard "shooting" method to find the normalizable mode to determine the 
onset of the instability, which is given by the smallest available baryon number density that leads 
normalizable unstable modes at a given temperature. 

First we consider the master equation (|27p . We assume the mode expansion /j = gi(U)e~' l ( ult+k: > xJ h 
Then, as explained in [5], the differential operator tijkdj has the eigenvalues ±k and where 
k = Vk i k i , and the equations of motion can therefore be diagonalized with respect to i indepen- 
dently of U. Note that the mode with zero eigenvalue turns out to be unphysical since it is in 
conflict with the Bianchi identity. Thus we diagonalize a« and drop the subscript i for gi(U) from 
now on, and these three are distinguished by the eigenvalue that appears in the equation of motion 

(EZJ). 

We are looking for the normalizable solutions with negative w 2 . Near the boundary, the asymp- 
totic solution is 

where the leading constant m part gives the nonnormalizable mode and v\J~^l 2 term is the nor- 
malizable mode, and m and v are constants of integration. We are going to tune k for given 
parameters to find a solution that has vanishing m. Now the connected D8-brane configuration is 
symmetric under 24 -H- — X4, and then two independent modes are either even or odd functions. 
This is equivalent to choose Neumann boundary condition {g(U c ) = l,g'(U c ) = 0} or Dirichlet one 
{g(U c ) = 0,g'(U c ) = 1} at the connecting point[6]. Among these choices, we have observed that 
the first choice is prone to be unstable for various choices of the parameters. When we consider the 
other set of the equations of motion, we also take the same two boundary conditions for ajj fields, 
while for y and ao fluctuations we need to take the instantons into account. Since we have put 
the wrapped D4-branes at U = U c and have the boundary action ([5]), thus we take the boundary 
conditions that do not change this boundary term imposed as the background. So we take Dirichlet 
boundary condition for y so that the position of D4 brane would not change. Since the conjugate 
variable of ao with respect to U is a constant of motion, we put Neumann boundary condition for 
ao, in order for this constant of motion, which is related to the instanton number density n^, not 

6 Note that in this parameter choice, A is not quite large and there exists a potential danger of having open string 
tachyons for too small I/[6]. Our choice avoids this problem. 
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g(^max) 




n b = 9.750 



Figure 5: The numerical cartoon indicates the dynamical instability at oj = to form the modulated 
phase in low temperature phase by tuning n&; namely, there exists a normalizable solution at critical 
ni,. The critical value of rib is around 9.726. 

to change. Finally, when we consider the instability for the QGP phase, we impose the in-falling 
boundary condition at the horizon, as usual. 

We first analyze the marginal case u 2 = 0, where an unstable mode would start to appear. 
In the low temperature phase, we tune the baryon number density rib to find the appearance of 
the instability. For the high temperature phase, we have another parameter Ut that gives the 
temperature T. 

The unstable mode appearing at the smallest rib = ^6(crit) can be considered to be the onset 
of the instability. The dynamical instability occurs for rib > n b(crit) a t a given temperature T. 
Using this criterion, we determine the phase boundary between the "nuclear matter phase" and 
the "modulated phase" in the T-rib plane. Note that we have not confirmed that this modulated 
phase stably exists, and we focus on drawing the phase boundaries. In Mathematica we implement 
a shooting method to look for the normalizable modes of g(U) which vanish at large U. The 
Mathematica code is like the following schematically, 

NDSolve[{EDM(k,w 2 = 0) of g,g[U c ] == l,g'[U c ] == 0}, g, {U, U c , U max }]. (30) 

Note that there are momentum dependences in the equation of motion. So we need to set up a do- 
loop to run over a range of momentum k and plot the diagram of k-g[U max ], where U max is chosen 
such that the sub-leading terms in g(U max ) can be dropped out and only the leading constant term 
m remains. 

Here we provide the plot to show how we look for the dynamical instabilities in the low temper- 
ature phase; the plots in the HT phase at a given temperature are similar. Fig. [5] shows that the 
critical value of rib at w 2 = is around rib ~ 9.726 since at this value there is a zero for g{U max ). 
That is to say we can find a normalizable solution at this critical value with u> 2 = 0. Repeating this 
procedure for both low and high temperature phases, including QGP phase in the high tempera- 
ture phase, the boundary to the the modulated phase (due to dynamical instability) is determined. 
Finally we translate these values of rib into the chemical potential in the corresponding phase to 
the phase boundaries for ji—T diagram. The result is summarized in Fig. [TJ 
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From our analysis, we see that rib arid n govern the emergence of the instability, and for a given 
k, which is determined by the choice of the parameters, we may choose a sufficiently large rib to 
find the onset of the instability 



Figure 6: The dispersion of k and —oj 2 for the normalizable unstable modes in the low temperature 
phase. The lower (blue) is for rib = 9-74 and the higher (red) is for rib = 9.80. 

To confirm that the normalizable solutions we obtained indeed indicate instability, we next 
check the k dependence of u(k). Fig. [6] is the plot of k dependence of —u(k) 2 at which the 
normalizable unstable solutions appear, for n& = 9.74 and rib = 9.80 in the low temperature phase. 
The horizontal axis is k and the vertical axis is -co 2 . It shows that unstable modes appear between 
two marginal values of k for rib > 9.726 as seen before. 

Finally, we also analyze the instability of the other three modes. They cannot be decoupled, 
and we then take the common oj = and k for all and search for the normalizable mode for one of 
them. For the instability to appear, all of the fluctuations have to be normalizable^- For the QGP 
phase, there seems no normalizable mode. In the nuclear matter phases, we found a normalizable 
mode with to = according to the combination of the boundary conditions, and it could be the 
onset of an instability. It appears at rather low baryon densities in general. However, the position 
of the onset depends very much on the choice of the boundary conditions, and we do not have 
the physical argument to fix this ambiguity at this moment. Therefore, we concentrate on the 
instability corresponding to fi modes we discussed, and do not include into our phase diagram 
these might-be onsets of the instabilities from the other modes. It will then be nice if one could 
perform the full analysis and discuss the instabilities regarding other modes. 

7 Nonnormalizable modes may become normalizable as the frequency and the momentum change, however, we do 
not perform the full analysis here, but just the preliminary ones. 




2.8 2.9 3 3.1 



k 



13 



5 Conclusion 



In this paper we study the dynamical instability of holographic QCD at finite density, which is 
usually difficult to attack by the conventional perturbative approach or the first principle lattice 
simulation due to the sign problem in the presence of finite chemical potential. One more advantage 
of holographic approach is to have the bulk geometric picture to illustrate the dynamical proper- 
ties of QCD. For example, both the thermodynamical and dynamical instabilities are due to the 
pulling force of the D4 baryon branes exerted on the probe D8-D8 meson branes. Our results are 
summarized in the phase diagram Fig. 1. The essential feature in Fig. 1 is the possible existence 
of the new modulated phase, which is missed by the homogeneity condition of thermodynamical 
consideration. 

A key ingredient for the appearance of such a dynamical instability is that the baryon density 
induces the electric field in the holographic D8-D8 brane, which couples to the QCD chiral current 
via the bulk Chern-Simons term. We find that the emergence of the instability is determined by the 
effective Chern-Simons coupling and the value of the baryon number density. By tuning the baryon 
number density high enough, a dynamical instability would appear, and this is consistent with the 
expectation from the previous large iV c analysis, at least for sufficient low temperatures. We have 
investigated the instabilities for several fluctuations on D8, and it should also be interesting to take 
the open string tachyon into account, which is dual to the chiral condensate and is an essential 
feature missed by the original Sakai-Sugimoto model, and to consider the dynamical instability 
with it. 

Finally, we are left with an open question about the treatment of the non-Abelian contribution 
to the DBI action. As already stated, it is technically difficult to consistently include the effect, 
especially including the back-reaction to the D8-brane configuration. Therefore in the paper, we 
instead took only its energy into the system as the D4-brane action on the internal sphere. Though 
there has been a lot of literature on holographic QCD with finite baryon density, complete treatment 
of the non-Abelian instantons seems to be still an open problem. It is therefore very important to 
overcome this problem and to find a consistent way to include the whole contribution in the future. 

Note added: During the preparation of the current version of the draft, we have been informed 
that Prof. Ooguri is also working on a similar problem about the instability in the QGP phase. 
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